Fix/prometheus metadata sorting #12112

Sinscerly · 2025-11-20T12:04:19Z

Description

This PR adds Prometheus metadata to all metrics types. Therefor Prometheus can include the TYPE and HELP information when scraping the endpoint.

See more details in #12110.

Fixes: #12110

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

I've tested the code change with a simplistic setup (didn't got all running, @abh1sar could you test it together more fully with your extra Prometheus metrics as my dev environment is still not working correctly)
I've successfully seen that the TYPE and HELP information is added to the nicely sorted list of metrics the exporter provides.

How did you try to break this feature and the system with this change?

I've enabled the Prometheus exporter and queried the endpoint multiple times to see the data came out like this:

# Cloudstack Prometheus Metrics
# HELP cloudstack_domain_limit_cpu_cores_total Total CPU core limit across all domains
# TYPE cloudstack_domain_limit_cpu_cores_total gauge
cloudstack_domain_limit_cpu_cores_total 0
# HELP cloudstack_domain_limit_memory_mibs_total Total memory limit in MiB across all domains
# TYPE cloudstack_domain_limit_memory_mibs_total gauge
cloudstack_domain_limit_memory_mibs_total 0
# HELP cloudstack_domain_resource_count Resource usage count per domain
# TYPE cloudstack_domain_resource_count gauge
cloudstack_domain_resource_count{domain="/", type="memory"} 0
cloudstack_domain_resource_count{domain="/", type="cpu"} 0
cloudstack_domain_resource_count{domain="/", type="gpu"} 0
cloudstack_domain_resource_count{domain="/", type="primary_storage"} 0

boring-cyborg · 2025-11-20T12:04:23Z

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
Be sure to read the CloudStack Coding Conventions.
Apache CloudStack is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/

DaanHoogland · 2025-11-21T11:07:36Z

@blueorangutan package

blueorangutan · 2025-11-21T11:08:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-11-21T11:16:16Z

Codecov Report

❌ Patch coverage is 0% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.56%. Comparing base (6dc259c) to head (8851dd9).
⚠️ Report is 86 commits behind head on main.

Files with missing lines	Patch %	Lines
...che/cloudstack/metrics/PrometheusExporterImpl.java	0.00%	43 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #12112   +/-   ##
=========================================
  Coverage     17.55%   17.56%           
- Complexity    15535    15541    +6     
=========================================
  Files          5911     5911           
  Lines        529359   529377   +18     
  Branches      64655    64656    +1     
=========================================
+ Hits          92949    92993   +44     
+ Misses       425952   425924   -28     
- Partials      10458    10460    +2

Flag	Coverage Δ
uitests	`3.58% <ø> (ø)`
unittests	`18.63% <0.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2025-11-21T12:25:55Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15807

DaanHoogland · 2025-11-24T10:31:10Z

@blueorangutan test

blueorangutan · 2025-11-24T10:32:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-11-24T11:17:01Z

[SF] Trillian Build Failed (tid-14851)

nvazquez · 2025-11-24T12:27:28Z

...egrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java

+            if (!item.name.equals(currentMetricName)) {
+                currentMetricName = item.name;
+                stringBuilder.append("# HELP ").append(currentMetricName).append(" ")
+                            .append(item.getHelp()).append("\n");


Hi @Sinscerly - I have a question around the metrics using tags in which you have modified the help text - if not mistaken the text won't be updated as the modification will happen after invoking item.toMetricsString() so at this line the previous help text will be displayed, is this correct?

Hi @nvazquez,

The HELP and TYPE only have to be added before each group of metrics. Only once for cloudstack_domain_resource_count, regardless of how many items with different tags share that metric name.

Does this clarify your question?

@Sinscerly sorry for the delay. My concern was only for the metrics that override the help field on the item.toMetricsString() call.

For example:
On the metrics iteration, when creating the metrics string for ItemHostCpu the string will always start with the default HELP text (Host CPU usage in MHz) despite having tags or not - I see in case the metric contains tags it should display Host CPU usage in MHz grouped by host tags. Please correct me if I'm wrong

Additionally, I think this if block if (!item.name.equals(currentMetricName)) can be removed

Hi @nvazquez,

Well the item ItemHostCpu returns two different metrics: cloudstack_host_cpu_usage_mhz_total and cloudstack_host_cpu_usage_mhz_total_by_tag. I just made sure the correct help line is outputted for that metric.

In the end all metrics are sorted and each unique help line is added. so this will give:

# HELP cloudstack_host_cpu_usage_mhz_total <help description> ... metrics for above type ... '' # HELP cloudstack_host_cpu_usage_mhz_total_by_tag <help description> ... metrics for above type

So it seemed logical for me to also change that. Although I'm doubting it will work, as the toMetricsString is called after adding in the HELP and TYPE I see. Although this would require some bigger rework of adding metrics I think, split it up in separate items.

Removing the if (!item.name.equals(currentMetricName)) will cause harm as you only want it once for every unique metric name. As you want:

# HELP cloudstack_domain_resource_count Resource usage count per domain # TYPE cloudstack_domain_resource_count gauge cloudstack_domain_resource_count{domain="/", type="memory"} 0 cloudstack_domain_resource_count{domain="/", type="cpu"} 0 cloudstack_domain_resource_count{domain="/", type="gpu"} 0 cloudstack_domain_resource_count{domain="/", type="primary_storage"} 0

And not the HELP and TYPE for every single metric item:

# HELP cloudstack_domain_resource_count Resource usage count per domain # TYPE cloudstack_domain_resource_count gauge cloudstack_domain_resource_count{domain="/", type="memory"} 0 # HELP cloudstack_domain_resource_count Resource usage count per domain # TYPE cloudstack_domain_resource_count gauge cloudstack_domain_resource_count{domain="/", type="cpu"} 0 # HELP cloudstack_domain_resource_count Resource usage count per domain # TYPE cloudstack_domain_resource_count gauge cloudstack_domain_resource_count{domain="/", type="gpu"} 0 # HELP cloudstack_domain_resource_count Resource usage count per domain # TYPE cloudstack_domain_resource_count gauge cloudstack_domain_resource_count{domain="/", type="primary_storage"} 0

Thanks @Sinscerly please ignore my second comment then :)

About the first one, do you think that can be addressed on a separate PR, leaving this PR as it is now?

blueorangutan · 2025-11-25T09:49:50Z

[SF] Trillian Build Failed (tid-14866)

blueorangutan · 2025-11-26T05:16:01Z

[SF] Trillian test result (tid-14870)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 49860 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12112-t14870-kvm-ol8.zip
Smoke tests completed. 150 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2025-11-26T08:34:38Z

@kiranchavala @NuxRo , can you guys have a look at this?

DaanHoogland · 2025-12-16T15:48:46Z

@Sinscerly , given comment #12112 (comment), will you work on this more or do you want this merged as is? cc @kiranchavala @NuxRo @nvazquez

NuxRo

tested briefly, LGTM, good job @Sinscerly

DaanHoogland · 2025-12-23T16:21:12Z

@kiranchavala @nvazquez @Sinscerly , will we merge as is and addresst the rest in further issues ?

Sinscerly added 2 commits November 20, 2025 12:24

fix prometheus metadata and sorted. Adds in TYPE and HELP

77092bc

fix random linting issue

8851dd9

DaanHoogland added this to the 4.23 milestone Nov 24, 2025

DaanHoogland added the status:needs-testing label Nov 24, 2025

nvazquez reviewed Nov 24, 2025

View reviewed changes

NuxRo approved these changes Dec 23, 2025

View reviewed changes

Fix/prometheus metadata sorting #12112

Are you sure you want to change the base?

Fix/prometheus metadata sorting #12112

Conversation

Sinscerly commented Nov 20, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

boring-cyborg bot commented Nov 20, 2025

Uh oh!

DaanHoogland commented Nov 21, 2025

Uh oh!

blueorangutan commented Nov 21, 2025

Uh oh!

codecov bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Nov 21, 2025

Uh oh!

DaanHoogland commented Nov 24, 2025

Uh oh!

blueorangutan commented Nov 24, 2025

Uh oh!

blueorangutan commented Nov 24, 2025

Uh oh!

nvazquez Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Sinscerly Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

nvazquez Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Sinscerly Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvazquez Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Nov 25, 2025

Uh oh!

blueorangutan commented Nov 26, 2025

Uh oh!

DaanHoogland commented Nov 26, 2025

Uh oh!

DaanHoogland commented Dec 16, 2025

Uh oh!

NuxRo left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Dec 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Nov 21, 2025 •

edited

Loading

Sinscerly Dec 15, 2025 •

edited

Loading